Repository navigation
Database: WAL, batched clears of deletion records, and giving free space back - #53
ChakraFusion wants to merge 4 commits into
Conversation
|
@ChakraFusion is attempting to deploy a commit to the sivadaboi's projects Team on Vercel. A member of the Team first needs to authorize it. |
1c9e890 to
f9b1bc8
Compare
|
Rebased onto today's |
|
Thanks, the WAL and batching changes look good. One request: could the one-time |
opensave.db had grown to 663 MB on one device, 262 MB of it pages that deleted rows had left: SQLite never shrinks the file on its own, and snapshots - each with a row per file of its save, a quarter of a million for one game - are merged, pruned and replaced all the time. The database is switched to incremental auto-vacuum, which takes one full VACUUM when it is opened (8 seconds for that database, 663 to 380 MB). After that its free pages are handed back a few at a time - when it is opened and hourly - so deleting a snapshot makes the file smaller. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Putting Project Zomboid back after its folder was emptied spent most of an hour in the database, the save untouched meanwhile: every file's deletion record was cleared with a statement of its own, a quarter of a million of them, and with SQLite's default rollback journal every one of them created, flushed and deleted a journal file. - Deletion records are cleared in one transaction per save location, on putting a save back and when files come back in a snapshot (ClearDeletedFiles): 50,000 take a third of a second. - The database runs in WAL mode with synchronous NORMAL: writes are appended to one log instead of each making a journal file of its own. The database stays whole whatever happens; a power cut can cost only the last moments' writes. Co-Authored-By: Claude Opus 5.5 <[email protected]>
One statement per path, even inside one transaction, took 10s for 50,000 records under the race detector on CI's runners, and the test's limit failed. 500 paths to a statement: 50,000 records in about 0.3s here. The test's limit is generous now (30s): what it guards against took minutes. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Asked in review: VACUUM inside store.Open made the first start after updating as long as rewriting the whole database, and it needs about the database's size free on its disk. - store.Open no longer rewrites anything. A new database gets incremental auto-vacuum for free: openDB asks for it first in the connection string, while the file is still empty. - A database from an older build is switched over by Store.EnableIncrementalVacuum, which the daemon runs once, two minutes after it has started, then hands free pages back as before (and hourly). It is not attempted without the database's size free next to it; the log says so, and it is tried again at the next start. - The log says when the one-time rewrite ran and how long it took. Test: a database created without auto-vacuum is left as it is by Open, switched over by EnableIncrementalVacuum, and not rewritten a second time. It fails on the previous commit, where Open did the VACUUM. Co-Authored-By: Claude Opus 5.5 <[email protected]>
f9b1bc8 to
c43626f
Compare
|
Done (c43626f):
Test: a database created without auto-vacuum is left alone by |
As offered in #21. Three commits:
VACUUMwhen the database is opened, then free pages handed back at open and hourly. On one PC, 663 MB became 380 MB.synchronous=NORMAL, and deletion records cleared in one transaction when a save is put back or files reappear. Putting a 240k-file save back had spent most of an hour creating and deleting a rollback journal per row.DELETE: 50,000 records in about 0.3 s, versus 10 s with one statement per path under the race detector.Nothing copies
opensave.dbdirectly; everything goes throughstore.Open, so the-walfile needs no special handling.Tests: the database uses WAL; compaction gives space back (using a table of the test's own); clearing 50,000 records is one transaction and fast.
🤖 Generated with Claude Code